Gradient Coding With Dynamic Clustering for Straggler-Tolerant Distributed Learning

Distributed implementations are crucial in speeding up large scale machine learning applications. gradient descent (GD) is widely employed to parallelize the task by distributing dataset across multiple workers. A significant performance bottleneck for per-iteration completion time distributed synchronous GD straggling Coded computation techniques have been introduced recently mitigate stragglers and speed iterations assigning redundant computations In this paper, we introduce a novel paradigm of dynamic coded computation, which assigns data workers acquire flexibility dynamically choose from among set possible codes depending on past behavior. particular, propose coding (GC) with clustering, called GC-DC, regulate number each cluster forming clusters at iteration. With time-correlated behavior, GC-DC adapts behavior over time; iteration, aims as uniformly based straggler For both homogeneous heterogeneous worker models, numerically show that provides improvements average without an increase communication load compared original GC scheme.